2024-09-24 00:15:14,536 [ 129273 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy (runner:42, check_args_and_update_paths) 2024-09-24 00:15:14,537 [ 129273 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration (runner:90, check_args_and_update_paths) 2024-09-24 00:15:14,537 [ 129273 ] INFO : src dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/src (runner:97, check_args_and_update_paths) 2024-09-24 00:15:14,537 [ 129273 ] INFO : base_configs_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration (runner:99, check_args_and_update_paths) 2024-09-24 00:15:14,537 [ 129273 ] INFO : Unknown image altinityinfra/dotnet-client (runner:357, ) Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_zqgugc --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_BASE_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_KERBERIZED_HADOOP_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_KERBEROS_KDC_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_MYSQL_JAVA_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_MYSQL_JS_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_MYSQL_PHP_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_NGINX_DAV_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_NEW_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_logs_level/test.py::test_check_client_logs_level test_mask_sensitive_info/test.py::test_backup_to_s3 test_mask_sensitive_info/test.py::test_create_alter_user test_mask_sensitive_info/test.py::test_create_database test_mask_sensitive_info/test.py::test_create_dictionary test_mask_sensitive_info/test.py::test_create_table test_mask_sensitive_info/test.py::test_encryption_functions test_mask_sensitive_info/test.py::test_on_cluster test_mask_sensitive_info/test.py::test_table_function_ways_to_call test_mask_sensitive_info/test.py::test_table_functions test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery test_merge_table_over_distributed/test.py::test_filtering test_merge_table_over_distributed/test.py::test_global_in test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_zookeeper/test.py::test_reload_zookeeper test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replace_partition/test.py::test_normal_work test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke -vvv" altinityinfra/integration-tests-runner:0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.2.0, repeat-0.9.3, order-1.0.1, xdist-3.5.0, random-0.2 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_reloading_storage_configuration/test.py::test_add_disk test_rocksdb_options/test.py::test_invalid_column_family_options test_merge_table_over_distributed/test.py::test_filtering test_multi_access_storage_role_management/test.py::test_quotas test_log_family_s3/test.py::test_log_family_s3[Log] test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_recovery_replica/test.py::test_choose_source_replica test_keeper_four_word_command/test.py::test_cmd_clrs test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_mask_sensitive_info/test.py::test_backup_to_s3 [gw8] [ 1%] PASSED test_merge_table_over_distributed/test.py::test_filtering test_merge_table_over_distributed/test.py::test_global_in [gw8] [ 2%] FAILED test_merge_table_over_distributed/test.py::test_global_in test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed [gw8] [ 3%] FAILED test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed [gw0] [ 4%] PASSED test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf [gw0] [ 5%] PASSED test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons [gw0] [ 6%] PASSED test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst [gw0] [ 7%] PASSED test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp [gw0] [ 8%] PASSED test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump [gw0] [ 9%] PASSED test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro [gw0] [ 10%] PASSED test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif [gw7] [ 11%] PASSED test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages [gw3] [ 12%] PASSED test_log_family_s3/test.py::test_log_family_s3[Log] test_log_family_s3/test.py::test_log_family_s3[StripeLog] [gw0] [ 13%] PASSED test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr [gw0] [ 14%] PASSED test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld [gw0] [ 15%] PASSED test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok [gw0] [ 16%] PASSED test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_session/test.py::test_session_close_shutdown [gw0] [ 17%] PASSED test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr [gw3] [ 18%] PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] test_log_family_s3/test.py::test_log_family_s3[TinyLog] [gw0] [ 19%] PASSED test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat [gw0] [ 20%] PASSED test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc [gw0] [ 21%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp [gw6] [ 22%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] [gw0] [ 23%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs [gw0] [ 24%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchs [gw3] [ 25%] PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] [gw5] [ 26%] PASSED test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options [gw6] [ 27%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] test_keeper_force_recovery_single_node/test.py::test_cluster_recovery [gw1] [ 28%] PASSED test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with [gw7] [ 29%] PASSED test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_row_policy/test.py::test_change_of_users_xml_changes_row_policies [gw6] [ 30%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] [gw2] [ 31%] PASSED test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy [gw6] [ 32%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] [gw7] [ 33%] PASSED test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles [gw6] [ 34%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw5] [ 35%] PASSED test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_column_family_options [gw2] [ 36%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config [gw6] [ 37%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw7] [ 38%] PASSED test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users [gw9] [ 39%] PASSED test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery [gw7] [ 40%] PASSED test_multi_access_storage_role_management/test.py::test_users [gw9] [ 41%] PASSED test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata [gw2] [ 42%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy [gw1] [ 43%] FAILED test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit [gw1] [ 43%] ERROR test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection test_keeper_nodes_move/test.py::test_node_move test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task [gw5] [ 44%] PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options [gw2] [ 45%] PASSED test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy [gw8] [ 46%] PASSED test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout [gw8] [ 47%] PASSED test_keeper_session/test.py::test_session_timeout test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch [gw1] [ 48%] FAILED test_row_policy/test.py::test_dcl_introspection [gw1] [ 48%] ERROR test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management [gw1] [ 49%] FAILED test_row_policy/test.py::test_dcl_management [gw1] [ 49%] ERROR test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml [gw2] [ 50%] PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works [gw5] [ 51%] PASSED test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options [gw5] [ 52%] PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options [gw5] [ 53%] PASSED test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options [gw5] [ 54%] PASSED test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options [gw5] [ 55%] PASSED test_rocksdb_options/test.py::test_valid_options test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings [gw2] [ 56%] PASSED test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk [gw1] [ 57%] FAILED test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml [gw1] [ 57%] ERROR test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy [gw1] [ 58%] FAILED test_row_policy/test.py::test_grant_create_row_policy [gw1] [ 58%] ERROR test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection [gw9] [ 59%] PASSED test_recovery_replica/test.py::test_update_metadata [gw4] [ 60%] PASSED test_mask_sensitive_info/test.py::test_backup_to_s3 test_mask_sensitive_info/test.py::test_create_alter_user [gw2] [ 61%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy [gw0] [ 62%] PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_part_uuid/test.py::test_part_uuid [gw4] [ 63%] PASSED test_mask_sensitive_info/test.py::test_create_alter_user test_mask_sensitive_info/test.py::test_create_database test_keeper_four_word_command/test_allow_list.py::test_allow_list [gw5] [ 64%] PASSED test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings [gw1] [ 65%] FAILED test_row_policy/test.py::test_introspection [gw1] [ 65%] ERROR test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join [gw4] [ 66%] PASSED test_mask_sensitive_info/test.py::test_create_database test_mask_sensitive_info/test.py::test_create_dictionary [gw2] [ 67%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy [gw4] [ 68%] PASSED test_mask_sensitive_info/test.py::test_create_dictionary test_mask_sensitive_info/test.py::test_create_table [gw8] [ 69%] PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_replication_without_zookeeper/test.py::test_startup_without_zookeeper [gw8] [ 70%] PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns [gw2] [ 71%] PASSED test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy [gw1] [ 72%] FAILED test_row_policy/test.py::test_join [gw1] [ 72%] ERROR test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines [gw0] [ 73%] PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list [gw2] [ 74%] PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_keeper_mntr_pressure/test.py::test_aggressive_mntr [gw4] [ 75%] PASSED test_mask_sensitive_info/test.py::test_create_table test_mask_sensitive_info/test.py::test_encryption_functions [gw1] [ 76%] FAILED test_row_policy/test.py::test_miscellaneous_engines [gw1] [ 76%] ERROR test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned [gw4] [ 77%] FAILED test_mask_sensitive_info/test.py::test_encryption_functions test_mask_sensitive_info/test.py::test_on_cluster [gw2] [ 78%] PASSED test_reloading_storage_configuration/test.py::test_update_disk [gw1] [ 79%] FAILED test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned [gw4] [ 80%] PASSED test_mask_sensitive_info/test.py::test_on_cluster test_mask_sensitive_info/test.py::test_table_function_ways_to_call [gw1] [ 80%] ERROR test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role [gw9] [ 81%] PASSED test_part_uuid/test.py::test_part_uuid [gw6] [ 82%] PASSED test_keeper_nodes_move/test.py::test_node_move [gw7] [ 83%] PASSED test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit [gw1] [ 84%] FAILED test_row_policy/test.py::test_policy_on_distributed_table_via_role [gw1] [ 84%] ERROR test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer [gw1] [ 85%] FAILED test_row_policy/test.py::test_reload_users_xml_by_timer test_replace_partition/test.py::test_normal_work [gw1] [ 85%] ERROR test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery [gw1] [ 86%] FAILED test_row_policy/test.py::test_row_policy_filter_with_subquery [gw1] [ 86%] ERROR test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke [gw5] [ 87%] PASSED test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_logs_level/test.py::test_check_client_logs_level test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery [gw1] [ 88%] FAILED test_row_policy/test.py::test_smoke test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster [gw1] [ 88%] ERROR test_row_policy/test.py::test_smoke [gw6] [ 89%] PASSED test_logs_level/test.py::test_check_client_logs_level [gw9] [ 90%] FAILED test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop [gw7] [ 91%] PASSED test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery test_read_temporary_tables_on_failure/test.py::test_different_versions [gw4] [ 92%] PASSED test_mask_sensitive_info/test.py::test_table_function_ways_to_call test_mask_sensitive_info/test.py::test_table_functions [gw8] [ 93%] PASSED test_replace_partition/test.py::test_normal_work [gw7] [ 94%] PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions [gw6] [ 95%] PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop [gw0] [ 96%] PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr [gw4] [ 97%] PASSED test_mask_sensitive_info/test.py::test_table_functions [gw3] [ 98%] FAILED test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table [gw3] [ 99%] PASSED test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table test_reload_zookeeper/test.py::test_reload_zookeeper [gw3] [100%] PASSED test_reload_zookeeper/test.py::test_reload_zookeeper ==================================== ERRORS ==================================== ______ ERROR at teardown of test_change_of_users_xml_changes_row_policies ______ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:15:48 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:15:48 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:15:48 [ 366 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node (cluster.py:3423, query) 2024-09-24 00:15:49 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node2_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:15:49 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node2_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:15:49 [ 366 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node2 (cluster.py:3423, query) 2024-09-24 00:15:49 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) 2024-09-24 00:15:49 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table2 on node (cluster.py:3423, query) 2024-09-24 00:15:50 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table3 on node (cluster.py:3423, query) 2024-09-24 00:15:50 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:15:50 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:15:50 [ 366 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node (cluster.py:3423, query) 2024-09-24 00:15:50 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node2_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:15:50 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node2_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:15:50 [ 366 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node2 (cluster.py:3423, query) 2024-09-24 00:15:50 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:16:07 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:07 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:07 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:07 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _________________ ERROR at teardown of test_dcl_introspection __________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:07 [ 366 ] DEBUG : Executing query SHOW POLICIES on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:16:18 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:18 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:18 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:18 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ___________________ ERROR at teardown of test_dcl_management ___________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:18 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:18 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:18 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:18 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:16:18 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:18 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:18 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:18 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _______ ERROR at teardown of test_dcl_users_with_policies_from_users_xml _______ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:18 [ 366 ] DEBUG : Executing query CREATE USER X on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:16:29 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:29 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:29 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:29 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ______________ ERROR at teardown of test_grant_create_row_policy _______________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:29 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:29 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:29 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:29 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:16:30 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:30 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:30 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:30 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ___________________ ERROR at teardown of test_introspection ____________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:30 [ 366 ] DEBUG : Executing query SELECT * from system.row_policies ORDER BY short_name, database, table on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:16:41 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:41 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:41 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:41 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ________________________ ERROR at teardown of test_join ________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:41 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 as t1 ANY LEFT JOIN mydb.filtered_table1 as t2 ON t1.a = t2.b on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:16:52 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:52 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:52 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:52 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _______________ ERROR at teardown of test_miscellaneous_engines ________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:52 [ 366 ] DEBUG : Executing query CREATE ROW POLICY OR REPLACE pC ON mydb.other_table FOR SELECT USING a = 1 TO default on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:17:02 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:02 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:02 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:02 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) __ ERROR at teardown of test_policy_from_users_xml_affects_only_user_assigned __ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:03 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:17:06 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:06 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:06 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:06 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ________ ERROR at teardown of test_policy_on_distributed_table_via_role ________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:06 [ 366 ] DEBUG : Executing query DROP TABLE IF EXISTS local_tbl on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:17:09 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:09 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:09 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:09 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _____________ ERROR at teardown of test_reload_users_xml_by_timer ______________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:09 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:09 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:09 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:09 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:17:09 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:09 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:09 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:09 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) __________ ERROR at teardown of test_row_policy_filter_with_subquery ___________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:10 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:10 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:10 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:10 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:17:10 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:10 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:10 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:10 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _______________________ ERROR at teardown of test_smoke ________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:10 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) --------------------------- Captured stderr teardown --------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:17:12 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:12 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:12 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:12 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) 2024-09-24 00:17:13 [ 366 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/.env', '--project-name', 'roottestrowpolicy', '--file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/node/docker-compose.yml', '--file', '/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/node2/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Stopping roottestrowpolicy_node2_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo3_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo2_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo1_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Stopping roottestrowpolicy_node2_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo1_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo2_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo3_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_row_policy/_instances_0/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_row_policy/_instances_0/node/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_row_policy/_instances_0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_row_policy/_instances_0/node2/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/.env', '--project-name', 'roottestrowpolicy', '--file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/node/docker-compose.yml', '--file', '/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/node2/docker-compose.yml', 'down', '--volumes'] (cluster.py:105, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_node2_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_node_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo3_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo2_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo1_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_node2_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_node_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo2_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo3_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo1_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Stderr:Removing network roottestrowpolicy_default (cluster.py:115, run_and_check) 2024-09-24 00:17:16 [ 366 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-09-24 00:17:16 [ 366 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.2.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.2 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 8 Running: 8 Paused: 0 Stopped: 0 Images: 42 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-119-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: bae9f20c8df0 ID: 23c7ffba-fa59-4a62-b9f4-6a037892fec4 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-09-24 00:17:16 [ 366 ] DEBUG : Docker networks for project roottestrowpolicy are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-09-24 00:17:17 [ 366 ] DEBUG : Docker containers for project roottestrowpolicy are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-09-24 00:17:17 [ 366 ] DEBUG : Docker volumes for project roottestrowpolicy are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-09-24 00:17:17 [ 366 ] DEBUG : Command:docker container list --all --filter name='^/roottestrowpolicy_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-09-24 00:17:17 [ 366 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-09-24 00:17:17 [ 366 ] DEBUG : No running containers for project: roottestrowpolicy (cluster.py:782, cleanup) 2024-09-24 00:17:17 [ 366 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-09-24 00:17:17 [ 366 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-09-24 00:17:17 [ 366 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-09-24 00:17:17 [ 366 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-09-24 00:17:17 [ 366 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-09-24 00:17:17 [ 366 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-09-24 00:17:17 [ 366 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-09-24 00:17:17 [ 366 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check) =================================== FAILURES =================================== ________________________________ test_global_in ________________________________ [gw8] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_global_in(started_cluster): assert ( node1.query( "SELECT val FROM distributed_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table)" ).rstrip() == "node2" ) > assert ( node1.query( "SELECT val FROM merge_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table)" ).rstrip() == "node2" ) test_merge_table_over_distributed/test.py:50: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 47, stderr: Received exception from server (version 23.8.16): E Code: 47. DB::Exception: Received from 172.16.3.3:9000. DB::Exception: Unknown expression or table expression identifier 'default._data_17601879111361243894_17988131190591442068' in scope SELECT local_table.val AS val FROM default.local_table WHERE local_table.id GLOBAL IN (default._data_17601879111361243894_17988131190591442068). Stack trace: E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type, std::type_identity::type, std::type_identity::type>, char const*&&, String&, String const&, String&&, String&&) @ 0x00000000312f96a7 in /usr/bin/clickhouse E 3. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5484: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000003122fc57 in /usr/bin/clickhouse E 4. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5688: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000003122a047 in /usr/bin/clickhouse E 5. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x00000000312633b2 in /usr/bin/clickhouse E 6. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x000000003122c32e in /usr/bin/clickhouse E 7. ./build_docker/./contrib/llvm-project/libcxx/include/vector:580: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000003121727c in /usr/bin/clickhouse E 8. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::QueryAnalysisPass::run(std::shared_ptr, std::shared_ptr) @ 0x0000000031211176 in /usr/bin/clickhouse E 9. ./build_docker/./src/Analyzer/QueryTreePassManager.cpp:0: DB::QueryTreePassManager::run(std::shared_ptr, unsigned long) @ 0x000000003120ca28 in /usr/bin/clickhouse E 10. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr const&, DB::SelectQueryOptions const&, std::shared_ptr const&, std::shared_ptr const&) @ 0x00000000315f3737 in /usr/bin/clickhouse E 11. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:161: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr const&, std::shared_ptr const&, DB::SelectQueryOptions const&) @ 0x00000000315f0029 in /usr/bin/clickhouse E 12. ./build_docker/./src/Processors/QueryPlan/DistributedCreateLocalPlan.cpp:92: DB::createLocalPlan(std::shared_ptr const&, DB::Block const&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long, unsigned long, unsigned long, std::shared_ptr, StrongTypedef, DB::UUIDTag>) @ 0x0000000035c036cb in /usr/bin/clickhouse E 13. ./build_docker/./src/Interpreters/ClusterProxy/SelectStreamFactory.cpp:126: DB::ClusterProxy::SelectStreamFactory::createForShard(DB::Cluster::ShardInfo const&, std::shared_ptr const&, DB::StorageID const&, std::shared_ptr const&, std::shared_ptr, std::vector>, std::allocator>>>&, std::vector>&, unsigned int, bool)::$_0::operator()() const @ 0x0000000032e1ddda in /usr/bin/clickhouse E 14. ./build_docker/./src/Interpreters/ClusterProxy/SelectStreamFactory.cpp:190: DB::ClusterProxy::SelectStreamFactory::createForShard(DB::Cluster::ShardInfo const&, std::shared_ptr const&, DB::StorageID const&, std::shared_ptr const&, std::shared_ptr, std::vector>, std::allocator>>>&, std::vector>&, unsigned int, bool) @ 0x0000000032e1b05c in /usr/bin/clickhouse E 15. ./build_docker/./src/Interpreters/ClusterProxy/executeQuery.cpp:220: DB::ClusterProxy::executeQuery(DB::QueryPlan&, DB::Block const&, DB::QueryProcessingStage::Enum, DB::StorageID const&, std::shared_ptr const&, DB::ClusterProxy::SelectStreamFactory&, Poco::Logger*, std::shared_ptr const&, std::shared_ptr, DB::SelectQueryInfo const&, std::shared_ptr const&, String const&, std::shared_ptr const&, std::function (unsigned long)>) @ 0x0000000032e27602 in /usr/bin/clickhouse E 16. ./build_docker/./src/Storages/StorageDistributed.cpp:802: DB::StorageDistributed::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x0000000032dd8cfa in /usr/bin/clickhouse E 17. ./build_docker/./src/Storages/StorageMerge.cpp:726: DB::ReadFromMerge::createSources(DB::QueryProcessingStage::Enum, unsigned long, DB::Block const&, std::tuple, std::shared_ptr, String> const&, std::vector>, DB::Block const&, std::shared_ptr, unsigned long, bool) @ 0x000000003304d131 in /usr/bin/clickhouse E 18. ./build_docker/./src/Storages/StorageMerge.cpp:496: DB::ReadFromMerge::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000033049014 in /usr/bin/clickhouse E 19. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c3fb12 in /usr/bin/clickhouse E 20. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c91273 in /usr/bin/clickhouse E 21. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000315f66b8 in /usr/bin/clickhouse E 22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:436: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000315f5ea7 in /usr/bin/clickhouse E 23. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e553d in /usr/bin/clickhouse E 24. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse E 25. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse E 26. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse E 27. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse E 28. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse E 29. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse E 30. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse E 31. ? @ 0x00007fb51ab72ac3 in ? E . (UNKNOWN_IDENTIFIER) E (query: SELECT val FROM merge_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table)) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:15:36 [ 387 ] DEBUG : Executing query SELECT val FROM distributed_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table) on node1 (cluster.py:3423, query) 2024-09-24 00:15:37 [ 387 ] DEBUG : Executing query SELECT val FROM merge_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table) on node1 (cluster.py:3423, query) ______________ test_select_table_name_from_merge_over_distributed ______________ [gw8] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_select_table_name_from_merge_over_distributed(started_cluster): node1.query("INSERT INTO local_table_2 VALUES (1, 'node1')") node2.query("INSERT INTO local_table_2 VALUES (2, 'node2')") > node1.query("select _table == 'distributed_table' from merge_table") test_merge_table_over_distributed/test.py:79: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 10, stderr: Received exception from server (version 23.8.16): E Code: 10. DB::Exception: Received from 172.16.3.3:9000. DB::Exception: Column `_table` not found in table default.merge_table (50c1e4a0-4790-4c05-8e23-2d743827aa3e). Stack trace: E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x000000000bd82fd6 in /usr/bin/clickhouse E 3. ./build_docker/./src/Storages/StorageSnapshot.cpp:142: DB::StorageSnapshot::getSampleBlockForColumns(std::vector> const&) const @ 0x00000000336432d8 in /usr/bin/clickhouse E 4. ./build_docker/./src/Planner/PlannerJoinTree.cpp:804: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000003165dca8 in /usr/bin/clickhouse E 5. ./build_docker/./src/Planner/PlannerJoinTree.cpp:1450: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000003164d9aa in /usr/bin/clickhouse E 6. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315feaf6 in /usr/bin/clickhouse E 7. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315f8e67 in /usr/bin/clickhouse E 8. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getSampleBlock() @ 0x00000000315f5a76 in /usr/bin/clickhouse E 9. ./build_docker/./src/Interpreters/getHeaderForProcessingStage.cpp:0: DB::getHeaderForProcessingStage(std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo const&, std::shared_ptr, DB::QueryProcessingStage::Enum) @ 0x0000000032118658 in /usr/bin/clickhouse E 10. ./build_docker/./src/Storages/StorageMerge.cpp:358: DB::StorageMerge::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x00000000330457d3 in /usr/bin/clickhouse E 11. ./build_docker/./src/Planner/PlannerJoinTree.cpp:756: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000003165ce97 in /usr/bin/clickhouse E 12. ./build_docker/./src/Planner/PlannerJoinTree.cpp:1450: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000003164d9aa in /usr/bin/clickhouse E 13. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315feaf6 in /usr/bin/clickhouse E 14. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315f8e67 in /usr/bin/clickhouse E 15. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x00000000315f6ab2 in /usr/bin/clickhouse E 16. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e467d in /usr/bin/clickhouse E 17. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse E 18. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse E 19. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse E 20. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse E 21. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse E 22. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse E 23. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse E 24. ? @ 0x00007fb51ab72ac3 in ? E 25. ? @ 0x00007fb51ac04850 in ? E . (NOT_FOUND_COLUMN_IN_BLOCK) E (query: select _table == 'distributed_table' from merge_table) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:15:37 [ 387 ] DEBUG : Executing query INSERT INTO local_table_2 VALUES (1, 'node1') on node1 (cluster.py:3423, query) 2024-09-24 00:15:37 [ 387 ] DEBUG : Executing query INSERT INTO local_table_2 VALUES (2, 'node2') on node2 (cluster.py:3423, query) 2024-09-24 00:15:37 [ 387 ] DEBUG : Executing query select _table == 'distributed_table' from merge_table on node1 (cluster.py:3423, query) --------------------------- Captured stderr teardown --------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:15:38 [ 387 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/.env', '--project-name', 'roottestmergetableoverdistributed', '--file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node2/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check) 2024-09-24 00:15:41 [ 387 ] DEBUG : Stderr:Stopping roottestmergetableoverdistributed_node2_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:15:41 [ 387 ] DEBUG : Stderr:Stopping roottestmergetableoverdistributed_node1_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:15:41 [ 387 ] DEBUG : Stderr:Stopping roottestmergetableoverdistributed_node1_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:15:41 [ 387 ] DEBUG : Stderr:Stopping roottestmergetableoverdistributed_node2_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:15:41 [ 387 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node1/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-09-24 00:15:41 [ 387 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node2/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-09-24 00:15:41 [ 387 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/.env', '--project-name', 'roottestmergetableoverdistributed', '--file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node2/docker-compose.yml', 'down', '--volumes'] (cluster.py:105, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Stderr:Removing roottestmergetableoverdistributed_node2_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Stderr:Removing roottestmergetableoverdistributed_node1_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Stderr:Removing roottestmergetableoverdistributed_node2_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Stderr:Removing roottestmergetableoverdistributed_node1_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Stderr:Removing network roottestmergetableoverdistributed_default (cluster.py:115, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-09-24 00:15:42 [ 387 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.2.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.2 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 29 Running: 29 Paused: 0 Stopped: 0 Images: 42 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-119-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: bae9f20c8df0 ID: 23c7ffba-fa59-4a62-b9f4-6a037892fec4 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-09-24 00:15:42 [ 387 ] DEBUG : Docker networks for project roottestmergetableoverdistributed are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-09-24 00:15:42 [ 387 ] DEBUG : Docker containers for project roottestmergetableoverdistributed are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-09-24 00:15:42 [ 387 ] DEBUG : Docker volumes for project roottestmergetableoverdistributed are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-09-24 00:15:42 [ 387 ] DEBUG : Command:docker container list --all --filter name='^/roottestmergetableoverdistributed_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-09-24 00:15:42 [ 387 ] DEBUG : No running containers for project: roottestmergetableoverdistributed (cluster.py:782, cleanup) 2024-09-24 00:15:42 [ 387 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-09-24 00:15:42 [ 387 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-09-24 00:15:42 [ 387 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-09-24 00:15:42 [ 387 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-09-24 00:15:42 [ 387 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-09-24 00:15:42 [ 387 ] DEBUG : Stdout:3 (cluster.py:113, run_and_check) ________________ test_change_of_users_xml_changes_row_policies _________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_change_of_users_xml_changes_row_policies(): copy_policy_xml("normal_filters.xml") assert node.query("SELECT * FROM mydb.filtered_table1") == TSV([[1, 0], [1, 1]]) assert node.query("SELECT * FROM mydb.filtered_table2") == TSV( [[0, 0, 0, 0], [0, 0, 6, 0]] ) assert node.query("SELECT * FROM mydb.filtered_table3") == TSV([[0, 1], [1, 0]]) copy_policy_xml("all_rows.xml") > assert node.query("SELECT * FROM mydb.filtered_table1") == TSV( [[0, 0], [0, 1], [1, 0], [1, 1]] ) test_row_policy/test.py:328: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 32, stderr: [node] 2024.09.24 00:15:51.036527 [ 299 ] BaseDaemon: ######################################## E [node] 2024.09.24 00:15:51.036674 [ 299 ] BaseDaemon: (version 23.8.16.41.altinitystable (altinity build), build id: E44628158B46C503106597583147763472F4E32F, git hash: 9992f5eba3591142ff413d31cb83a7bf4f0e2553) (from thread 8) (query_id: 1a866cf2-5f2b-4f51-8ebb-167c5307adeb) (query: SELECT * FROM mydb.filtered_table1) Received signal Segmentation fault (11) E [node] 2024.09.24 00:15:51.036754 [ 299 ] BaseDaemon: Address: NULL pointer. Access: read. Address not mapped to object. E [node] 2024.09.24 00:15:51.036814 [ 299 ] BaseDaemon: Stack trace: 0x0000563b5b2b59cd 0x0000563b5ba8e615 0x00007f45c93bd520 0x0000563b6edfc97c 0x0000563b6f28cf27 0x0000563b6f2b0bee 0x0000563b6f2a49aa 0x0000563b6f255af6 0x0000563b6f24fe67 0x0000563b6f24dab2 0x0000563b6fd3b67d 0x0000563b6fd33301 0x0000563b72b242fd 0x0000563b72b610d0 0x0000563b7b90308f 0x0000563b7b903e57 0x0000563b7bde367c 0x0000563b7bddc728 0x00007f45c940fac3 0x00007f45c94a1850 E [node] 2024.09.24 00:15:51.124575 [ 299 ] BaseDaemon: 0.1. inlined from ./build_docker/./src/Common/StackTrace.cpp:286: StackTrace::tryCapture() E [node] 2024.09.24 00:15:51.124678 [ 299 ] BaseDaemon: 0. ./build_docker/./src/Common/StackTrace.cpp:258: StackTrace::StackTrace(ucontext_t const&) @ 0x000000001d65e9cd in /usr/bin/clickhouse E [node] 2024.09.24 00:15:51.405493 [ 299 ] BaseDaemon: 1.1. inlined from ./build_docker/./src/IO/WriteHelpers.h:88: void DB::writePODBinary(int const&, DB::WriteBuffer&) E [node] 2024.09.24 00:15:51.405635 [ 299 ] BaseDaemon: 1.2. inlined from ./build_docker/./src/IO/WriteHelpers.h:881: void DB::writeBinary(int const&, DB::WriteBuffer&) E [node] 2024.09.24 00:15:51.405701 [ 299 ] BaseDaemon: 1. ./build_docker/./src/Daemon/BaseDaemon.cpp:152: signalHandler(int, siginfo_t*, void*) @ 0x000000001de37615 in /usr/bin/clickhouse E [node] 2024.09.24 00:15:51.405768 [ 299 ] BaseDaemon: 2. ? @ 0x00007f45c93bd520 in ? E [node] 2024.09.24 00:15:51.769554 [ 299 ] BaseDaemon: 3.1. inlined from ./build_docker/./src/Analyzer/QueryTreeBuilder.cpp:123: QueryTreeBuilder E [node] 2024.09.24 00:15:51.769640 [ 299 ] BaseDaemon: 3. ./build_docker/./src/Analyzer/QueryTreeBuilder.cpp:1029: DB::buildQueryTree(std::shared_ptr, std::shared_ptr) @ 0x00000000311a597c in /usr/bin/clickhouse E [node] 2024.09.24 00:15:52.159058 [ 299 ] BaseDaemon: 4. ./build_docker/./src/Planner/Utils.cpp:0: DB::buildFilterInfo(std::shared_ptr, std::shared_ptr const&, std::shared_ptr&, std::unordered_set, std::equal_to, std::allocator>) @ 0x0000000031635f27 in /usr/bin/clickhouse E [node] 2024.09.24 00:15:52.837371 [ 299 ] BaseDaemon: 5.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: ~__hash_table E [node] 2024.09.24 00:15:52.837501 [ 299 ] BaseDaemon: 5.2. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/unordered_set:615: ~unordered_set E [node] 2024.09.24 00:15:52.837580 [ 299 ] BaseDaemon: 5.3. inlined from ./build_docker/./src/Planner/PlannerJoinTree.cpp:447: DB::(anonymous namespace)::buildRowPolicyFilterIfNeeded(std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr&) E [node] 2024.09.24 00:15:52.837638 [ 299 ] BaseDaemon: 5. ./build_docker/./src/Planner/PlannerJoinTree.cpp:730: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x0000000031659bee in /usr/bin/clickhouse E [node] 2024.09.24 00:15:53.491839 [ 299 ] BaseDaemon: 6. ./build_docker/./src/Planner/PlannerJoinTree.cpp:1450: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000003164d9aa in /usr/bin/clickhouse E [node] 2024.09.24 00:15:53.819307 [ 299 ] BaseDaemon: 7. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315feaf6 in /usr/bin/clickhouse E [node] 2024.09.24 00:15:54.077722 [ 299 ] BaseDaemon: 8. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315f8e67 in /usr/bin/clickhouse E [node] 2024.09.24 00:15:54.193809 [ 299 ] BaseDaemon: 9.1. inlined from ./build_docker/./src/Planner/Planner.h:44: DB::Planner::getQueryPlan() E [node] 2024.09.24 00:15:54.193905 [ 299 ] BaseDaemon: 9. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:235: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x00000000315f6ab2 in /usr/bin/clickhouse E [node] 2024.09.24 00:15:54.684374 [ 299 ] BaseDaemon: 10. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e467d in /usr/bin/clickhouse E [node] 2024.09.24 00:15:55.141282 [ 299 ] BaseDaemon: 11. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse E [node] 2024.09.24 00:15:55.510525 [ 299 ] BaseDaemon: 12. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse E [node] 2024.09.24 00:15:56.119078 [ 299 ] BaseDaemon: 13. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse E [node] 2024.09.24 00:15:56.139782 [ 299 ] BaseDaemon: 14. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse E [node] 2024.09.24 00:15:56.170806 [ 299 ] BaseDaemon: 15.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: std::default_delete::operator()[abi:v15000](Poco::Net::TCPServerConnection*) const E [node] 2024.09.24 00:15:56.170952 [ 299 ] BaseDaemon: 15.2. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305: std::unique_ptr>::reset[abi:v15000](Poco::Net::TCPServerConnection*) E [node] 2024.09.24 00:15:56.171017 [ 299 ] BaseDaemon: 15.3. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259: ~unique_ptr E [node] 2024.09.24 00:15:56.171079 [ 299 ] BaseDaemon: 15. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:116: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse E [node] 2024.09.24 00:15:56.213528 [ 299 ] BaseDaemon: 16. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse E [node] 2024.09.24 00:15:56.245246 [ 299 ] BaseDaemon: 17. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse E [node] 2024.09.24 00:15:56.245357 [ 299 ] BaseDaemon: 18. ? @ 0x00007f45c940fac3 in ? E [node] 2024.09.24 00:15:56.245399 [ 299 ] BaseDaemon: 19. ? @ 0x00007f45c94a1850 in ? E [node] 2024.09.24 00:15:57.704054 [ 299 ] BaseDaemon: Integrity check of the executable successfully passed (checksum: 1E52029798AD9E6464A7305FA1863306) E [node] 2024.09.24 00:16:02.163330 [ 299 ] BaseDaemon: You are using an Altinity Stable Build. Please log issues at https://github.com/Altinity/ClickHouse/issues. Thank you! E [node] 2024.09.24 00:16:02.163619 [ 299 ] BaseDaemon: Changed settings: any_join_distinct_right_table_keys = true, allow_experimental_analyzer = true, optimize_on_insert = false E Error on processing query: Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from 172.16.7.5:9000. (ATTEMPT_TO_READ_AFTER_EOF), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /clickhouse E 2. DB::Exception::Exception(int, char const (&) [26]) @ 0x000000000c5e7649 in /clickhouse E 3. ./build_docker/./src/IO/VarInt.cpp:13: DB::throwReadAfterEOF() @ 0x000000001d755ffe in /clickhouse E 4. ./build_docker/./src/QueryPipeline/ProfileInfo.h:18: DB::Connection::receivePacket() @ 0x0000000034c0f183 in /clickhouse E 5. ./build_docker/./src/Client/ClientBase.cpp:1063: DB::ClientBase::receiveAndProcessPacket(std::shared_ptr, bool) @ 0x0000000034b83093 in /clickhouse E 6. ./build_docker/./src/Client/ClientBase.cpp:1038: DB::ClientBase::receiveResult(std::shared_ptr, int, bool) @ 0x0000000034b82546 in /clickhouse E 7. ./build_docker/./src/Client/ClientBase.cpp:0: DB::ClientBase::processOrdinaryQuery(String const&, std::shared_ptr) @ 0x0000000034b7fb96 in /clickhouse E 8. ./build_docker/./src/Client/ClientBase.cpp:0: DB::ClientBase::processParsedSingleQuery(String const&, String const&, std::shared_ptr, std::optional, bool) @ 0x0000000034b7b6aa in /clickhouse E 9. ./build_docker/./src/Client/ClientBase.cpp:2053: DB::ClientBase::executeMultiQuery(String const&) @ 0x0000000034b99c40 in /clickhouse E 10. ./build_docker/./src/Client/ClientBase.cpp:0: DB::ClientBase::processQueryText(String const&) @ 0x0000000034b9cd54 in /clickhouse E 11. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::ClientBase::runNonInteractive() @ 0x0000000034ba72f9 in /clickhouse E 12. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001d9fb5dc in /clickhouse E 13. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dce6207 in /clickhouse E 14. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da2dd7c in /clickhouse E 15. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd67eb5 in /clickhouse E 16. ? @ 0x00007f278a578d90 in ? E 17. ? @ 0x00007f278a578e40 in ? E 18. _start @ 0x000000000bc8f12e in /clickhouse E (version 23.8.16.41.altinitystable (altinity build)) E (query: SELECT * FROM mydb.filtered_table1) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:15:48 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:15:48 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:15:48 [ 366 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node (cluster.py:3423, query) 2024-09-24 00:15:49 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node2_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:15:49 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node2_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:15:49 [ 366 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node2 (cluster.py:3423, query) 2024-09-24 00:15:49 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) 2024-09-24 00:15:49 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table2 on node (cluster.py:3423, query) 2024-09-24 00:15:50 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table3 on node (cluster.py:3423, query) 2024-09-24 00:15:50 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:15:50 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:15:50 [ 366 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node (cluster.py:3423, query) 2024-09-24 00:15:50 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node2_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:15:50 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node2_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:15:50 [ 366 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node2 (cluster.py:3423, query) 2024-09-24 00:15:50 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) ____________________________ test_dcl_introspection ____________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_dcl_introspection(): > assert node.query("SHOW POLICIES") == TSV( [ "another ON mydb.local", "default ON mydb.filtered_table1", "default ON mydb.filtered_table2", "default ON mydb.filtered_table3", ] ) test_row_policy/test.py:476: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.7.5:9000 (172.16.7.5:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x0000000034c14241 in /clickhouse E 3. ./build_docker/./src/Client/Connection.cpp:218: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x0000000034bf347f in /clickhouse E 4. ./build_docker/./src/Client/Connection.cpp:423: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x0000000034bf9d1b in /clickhouse E 5. ./build_docker/./programs/client/Client.cpp:446: DB::Client::connect() @ 0x000000001d9fe9a2 in /clickhouse E 6. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001d9fb33e in /clickhouse E 7. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dce6207 in /clickhouse E 8. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da2dd7c in /clickhouse E 9. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd67eb5 in /clickhouse E 10. ? @ 0x00007f2e0c87fd90 in ? E 11. ? @ 0x00007f2e0c87fe40 in ? E 12. _start @ 0x000000000bc8f12e in /clickhouse helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:07 [ 366 ] DEBUG : Executing query SHOW POLICIES on node (cluster.py:3423, query) _____________________________ test_dcl_management ______________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_dcl_management(): > copy_policy_xml("no_filters.xml") test_row_policy/test.py:603: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:18 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:18 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:18 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:18 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _________________ test_dcl_users_with_policies_from_users_xml __________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_dcl_users_with_policies_from_users_xml(): > node.query("CREATE USER X") test_row_policy/test.py:732: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.7.5:9000 (172.16.7.5:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x0000000034c14241 in /clickhouse E 3. ./build_docker/./src/Client/Connection.cpp:218: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x0000000034bf347f in /clickhouse E 4. ./build_docker/./src/Client/Connection.cpp:423: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x0000000034bf9d1b in /clickhouse E 5. ./build_docker/./programs/client/Client.cpp:446: DB::Client::connect() @ 0x000000001d9fe9a2 in /clickhouse E 6. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001d9fb33e in /clickhouse E 7. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dce6207 in /clickhouse E 8. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da2dd7c in /clickhouse E 9. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd67eb5 in /clickhouse E 10. ? @ 0x00007fd1f13aed90 in ? E 11. ? @ 0x00007fd1f13aee40 in ? E 12. _start @ 0x000000000bc8f12e in /clickhouse helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:18 [ 366 ] DEBUG : Executing query CREATE USER X on node (cluster.py:3423, query) _________________________ test_grant_create_row_policy _________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_grant_create_row_policy(): > copy_policy_xml("no_filters.xml") test_row_policy/test.py:635: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:29 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:16:29 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:16:29 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:16:29 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ______________________________ test_introspection ______________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_introspection(): policies = [ [ "another ON mydb.local", "another", "mydb", "local", "5b23c389-7e18-06bf-a6bc-dd1afbbc0a97", "users_xml", "a = 1", 0, 0, "['another']", "[]", ], [ "default ON mydb.filtered_table1", "default", "mydb", "filtered_table1", "9e8a8f62-4965-2b5e-8599-57c7b99b3549", "users_xml", "a = 1", 0, 0, "['default']", "[]", ], [ "default ON mydb.filtered_table2", "default", "mydb", "filtered_table2", "cffae79d-b9bf-a2ef-b798-019c18470b25", "users_xml", "a + b < 1 or c - d > 5", 0, 0, "['default']", "[]", ], [ "default ON mydb.filtered_table3", "default", "mydb", "filtered_table3", "12fc5cef-e3da-3940-ec79-d8be3911f42b", "users_xml", "c = 1", 0, 0, "['default']", "[]", ], ] > assert node.query( "SELECT * from system.row_policies ORDER BY short_name, database, table" ) == TSV(policies) test_row_policy/test.py:470: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.7.5:9000 (172.16.7.5:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x0000000034c14241 in /clickhouse E 3. ./build_docker/./src/Client/Connection.cpp:218: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x0000000034bf347f in /clickhouse E 4. ./build_docker/./src/Client/Connection.cpp:423: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x0000000034bf9d1b in /clickhouse E 5. ./build_docker/./programs/client/Client.cpp:446: DB::Client::connect() @ 0x000000001d9fe9a2 in /clickhouse E 6. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001d9fb33e in /clickhouse E 7. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dce6207 in /clickhouse E 8. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da2dd7c in /clickhouse E 9. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd67eb5 in /clickhouse E 10. ? @ 0x00007f13efbf3d90 in ? E 11. ? @ 0x00007f13efbf3e40 in ? E 12. _start @ 0x000000000bc8f12e in /clickhouse helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:30 [ 366 ] DEBUG : Executing query SELECT * from system.row_policies ORDER BY short_name, database, table on node (cluster.py:3423, query) __________________________________ test_join ___________________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_join(): > assert node.query( "SELECT * FROM mydb.filtered_table1 as t1 ANY LEFT JOIN mydb.filtered_table1 as t2 ON t1.a = t2.b" ) == TSV([[1, 0, 1, 1], [1, 1, 1, 1]]) test_row_policy/test.py:126: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.7.5:9000 (172.16.7.5:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x0000000034c14241 in /clickhouse E 3. ./build_docker/./src/Client/Connection.cpp:218: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x0000000034bf347f in /clickhouse E 4. ./build_docker/./src/Client/Connection.cpp:423: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x0000000034bf9d1b in /clickhouse E 5. ./build_docker/./programs/client/Client.cpp:446: DB::Client::connect() @ 0x000000001d9fe9a2 in /clickhouse E 6. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001d9fb33e in /clickhouse E 7. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dce6207 in /clickhouse E 8. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da2dd7c in /clickhouse E 9. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd67eb5 in /clickhouse E 10. ? @ 0x00007f4459731d90 in ? E 11. ? @ 0x00007f4459731e40 in ? E 12. _start @ 0x000000000bc8f12e in /clickhouse helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:41 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 as t1 ANY LEFT JOIN mydb.filtered_table1 as t2 ON t1.a = t2.b on node (cluster.py:3423, query) __________________________ test_miscellaneous_engines __________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_miscellaneous_engines(): > node.query( "CREATE ROW POLICY OR REPLACE pC ON mydb.other_table FOR SELECT USING a = 1 TO default" ) test_row_policy/test.py:814: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.7.5:9000 (172.16.7.5:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x0000000034c14241 in /clickhouse E 3. ./build_docker/./src/Client/Connection.cpp:218: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x0000000034bf347f in /clickhouse E 4. ./build_docker/./src/Client/Connection.cpp:423: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x0000000034bf9d1b in /clickhouse E 5. ./build_docker/./programs/client/Client.cpp:446: DB::Client::connect() @ 0x000000001d9fe9a2 in /clickhouse E 6. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001d9fb33e in /clickhouse E 7. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dce6207 in /clickhouse E 8. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da2dd7c in /clickhouse E 9. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd67eb5 in /clickhouse E 10. ? @ 0x00007f2a59489d90 in ? E 11. ? @ 0x00007f2a59489e40 in ? E 12. _start @ 0x000000000bc8f12e in /clickhouse helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:52 [ 366 ] DEBUG : Executing query CREATE ROW POLICY OR REPLACE pC ON mydb.other_table FOR SELECT USING a = 1 TO default on node (cluster.py:3423, query) __________________________ test_encryption_functions ___________________________ [gw4] linux -- Python 3.10.12 /usr/bin/python3 def test_encryption_functions(): plaintext = new_password() cipher = new_password() key = new_password(32) iv8 = new_password(8) iv16 = new_password(16) add = new_password() encryption_functions = [ f"encrypt('aes-256-ofb', '{plaintext}', '{key}')", f"encrypt('aes-256-ofb', '{plaintext}', '{key}', '{iv16}')", f"encrypt('aes-256-gcm', '{plaintext}', '{key}', '{iv8}')", f"encrypt('aes-256-gcm', '{plaintext}', '{key}', '{iv8}', '{add}')", f"decrypt('aes-256-ofb', '{cipher}', '{key}', '{iv16}')", f"aes_encrypt_mysql('aes-256-ofb', '{plaintext}', '{key}', '{iv16}')", f"aes_decrypt_mysql('aes-256-ofb', '{cipher}', '{key}', '{iv16}')", f"tryDecrypt('aes-256-ofb', '{cipher}', '{key}', '{iv16}')", ] for encryption_function in encryption_functions: node.query(f"SELECT {encryption_function}") > check_logs( must_contain=[ "SELECT encrypt('aes-256-ofb', '[HIDDEN]')", "SELECT encrypt('aes-256-gcm', '[HIDDEN]')", "SELECT decrypt('aes-256-ofb', '[HIDDEN]')", "SELECT aes_encrypt_mysql('aes-256-ofb', '[HIDDEN]')", "SELECT aes_decrypt_mysql('aes-256-ofb', '[HIDDEN]')", "SELECT tryDecrypt('aes-256-ofb', '[HIDDEN]')", ], must_not_contain=[plaintext, cipher, key, iv8, iv16, add], ) test_mask_sensitive_info/test.py:381: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_mask_sensitive_info/test.py:46: in check_logs assert not node.contains_in_log(escaped_str) helpers/cluster.py:3863: in contains_in_log result = self.exec_in_container( helpers/cluster.py:3840: in exec_in_container return self.cluster.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "7RYY02WF4MW9JS7O" /var/log/clickhouse-server/clickhouse-server.log || true'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) > out = res.stdout.decode("utf-8") E UnicodeDecodeError: 'utf-8' codec can't decode byte 0xeb in position 152: invalid continuation byte helpers/cluster.py:109: UnicodeDecodeError ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:01 [ 375 ] DEBUG : Executing query SELECT encrypt('aes-256-ofb', '7RYY02WF4MW9JS7O', '1D8GIW84QQXLK6ZIIXCOHW2QJNVH0YOS') on node (cluster.py:3423, query) 2024-09-24 00:17:01 [ 375 ] DEBUG : Executing query SELECT encrypt('aes-256-ofb', '7RYY02WF4MW9JS7O', '1D8GIW84QQXLK6ZIIXCOHW2QJNVH0YOS', 'PNVIOVL6GD21ZRBD') on node (cluster.py:3423, query) 2024-09-24 00:17:02 [ 375 ] DEBUG : Executing query SELECT encrypt('aes-256-gcm', '7RYY02WF4MW9JS7O', '1D8GIW84QQXLK6ZIIXCOHW2QJNVH0YOS', 'UPXAU3JX') on node (cluster.py:3423, query) 2024-09-24 00:17:02 [ 375 ] DEBUG : Executing query SELECT encrypt('aes-256-gcm', '7RYY02WF4MW9JS7O', '1D8GIW84QQXLK6ZIIXCOHW2QJNVH0YOS', 'UPXAU3JX', 'YB9Y0KSEVAU6XYDP') on node (cluster.py:3423, query) 2024-09-24 00:17:02 [ 375 ] DEBUG : Executing query SELECT decrypt('aes-256-ofb', 'RPGSRFDB0BUK8DU4', '1D8GIW84QQXLK6ZIIXCOHW2QJNVH0YOS', 'PNVIOVL6GD21ZRBD') on node (cluster.py:3423, query) 2024-09-24 00:17:02 [ 375 ] DEBUG : Executing query SELECT aes_encrypt_mysql('aes-256-ofb', '7RYY02WF4MW9JS7O', '1D8GIW84QQXLK6ZIIXCOHW2QJNVH0YOS', 'PNVIOVL6GD21ZRBD') on node (cluster.py:3423, query) 2024-09-24 00:17:02 [ 375 ] DEBUG : Executing query SELECT aes_decrypt_mysql('aes-256-ofb', 'RPGSRFDB0BUK8DU4', '1D8GIW84QQXLK6ZIIXCOHW2QJNVH0YOS', 'PNVIOVL6GD21ZRBD') on node (cluster.py:3423, query) 2024-09-24 00:17:03 [ 375 ] DEBUG : Executing query SELECT tryDecrypt('aes-256-ofb', 'RPGSRFDB0BUK8DU4', '1D8GIW84QQXLK6ZIIXCOHW2QJNVH0YOS', 'PNVIOVL6GD21ZRBD') on node (cluster.py:3423, query) 2024-09-24 00:17:03 [ 375 ] DEBUG : Executing query SYSTEM FLUSH LOGS on node (cluster.py:3423, query) 2024-09-24 00:17:03 [ 375 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT encrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:03 [ 375 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT encrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:03 [ 375 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.09.24 00:17:01.724095 [ 9 ] {5725388b-f925-4fee-a322-b9efc5613e97} executeQuery: (from 172.16.4.1:57794) SELECT encrypt('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-09-24 00:17:03 [ 375 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.09.24 00:17:01.935567 [ 9 ] {e62d530b-b63b-4bb4-b63b-2f29dc588946} executeQuery: (from 172.16.4.1:57808) SELECT encrypt('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-09-24 00:17:03 [ 375 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT encrypt(\'aes-256-gcm\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:03 [ 375 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT encrypt(\'aes-256-gcm\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:03 [ 375 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.09.24 00:17:02.155770 [ 9 ] {4b8e3da7-1e08-4a55-90e4-b2edfd5ecab6} executeQuery: (from 172.16.4.1:57820) SELECT encrypt('aes-256-gcm', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-09-24 00:17:03 [ 375 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.09.24 00:17:02.367073 [ 9 ] {a941b4a3-3fe6-470f-8ca6-a3dcc993b97b} executeQuery: (from 172.16.4.1:57822) SELECT encrypt('aes-256-gcm', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-09-24 00:17:03 [ 375 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT decrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:03 [ 375 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT decrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:03 [ 375 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.09.24 00:17:02.582440 [ 9 ] {5486b073-cdce-47d9-9ad6-a10eb39904e2} executeQuery: (from 172.16.4.1:57830) SELECT decrypt('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-09-24 00:17:03 [ 375 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT aes_encrypt_mysql(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:03 [ 375 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT aes_encrypt_mysql(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:04 [ 375 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.09.24 00:17:02.801672 [ 9 ] {d824a510-b62a-447c-bddf-3f7dfc9da94e} executeQuery: (from 172.16.4.1:57834) SELECT aes_encrypt_mysql('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-09-24 00:17:04 [ 375 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT aes_decrypt_mysql(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:04 [ 375 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT aes_decrypt_mysql(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:04 [ 375 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.09.24 00:17:03.024406 [ 9 ] {6cb7d25e-4469-443a-b3c6-e5b32bf485f6} executeQuery: (from 172.16.4.1:57844) SELECT aes_decrypt_mysql('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-09-24 00:17:04 [ 375 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT tryDecrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:04 [ 375 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT tryDecrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:04 [ 375 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.09.24 00:17:03.232428 [ 9 ] {3ec0cd9c-0661-4f7a-b244-aabb23e1b41d} executeQuery: (from 172.16.4.1:57846) SELECT tryDecrypt('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-09-24 00:17:04 [ 375 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "7RYY02WF4MW9JS7O" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:04 [ 375 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "7RYY02WF4MW9JS7O" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) ____________ test_policy_from_users_xml_affects_only_user_assigned _____________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_policy_from_users_xml_affects_only_user_assigned(): > assert node.query("SELECT * FROM mydb.filtered_table1") == TSV([[1, 0], [1, 1]]) test_row_policy/test.py:208: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.7.5:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:03 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) __________________ test_policy_on_distributed_table_via_role ___________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_policy_on_distributed_table_via_role(): > node.query("DROP TABLE IF EXISTS local_tbl") test_row_policy/test.py:859: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.7.5:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:06 [ 366 ] DEBUG : Executing query DROP TABLE IF EXISTS local_tbl on node (cluster.py:3423, query) ________________________ test_reload_users_xml_by_timer ________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_reload_users_xml_by_timer(): > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:381: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:09 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:09 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:09 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:09 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _____________________ test_row_policy_filter_with_subquery _____________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_row_policy_filter_with_subquery(): > copy_policy_xml("no_filters.xml") test_row_policy/test.py:891: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:10 [ 366 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-09-24 00:17:10 [ 366 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-09-24 00:17:10 [ 366 ] DEBUG : Stderr:Error response from daemon: Container 665356facb0d9db6b70e34da0b3cddc90abcd792c0bb4fffb35125aed0bd4f6f is not running (cluster.py:115, run_and_check) 2024-09-24 00:17:10 [ 366 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) __________________________________ test_smoke __________________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_smoke(): > assert node.query("SELECT * FROM mydb.filtered_table1") == TSV([[1, 0], [1, 1]]) test_row_policy/test.py:98: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.7.5:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:10 [ 366 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) _______________________ test_default_database_on_cluster _______________________ [gw9] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_default_database_on_cluster(started_cluster): ch1.query( database="test_default_database", sql="CREATE TABLE test_local_table ENGINE MergeTree PARTITION BY i ORDER BY tuple() SETTINGS max_partitions_to_read = 1 AS SELECT arrayJoin([1, 2]) i;", ) > assert ( ch2.query( sql="SELECT * FROM remote('ch1:9000', test_default_database, test_local_table) ORDER BY i FORMAT TSV SETTINGS max_partitions_to_read = 0;", ) == "1\n2\n" ) test_passing_max_partitions_to_read_remotely/test.py:26: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 81, stderr: Received exception from server (version 23.8.16): E Code: 81. DB::Exception: Received from 172.16.23.3:9000. DB::Exception: Database test_default_database does not exist. Stack trace: E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e077d33 in /usr/bin/clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5c5dd0 in /usr/bin/clickhouse E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x000000000bd7e9db in /usr/bin/clickhouse E 3. ./build_docker/./src/Interpreters/DatabaseCatalog.cpp:650: DB::DatabaseCatalog::getDatabase(String const&) const @ 0x000000002fb2a082 in /usr/bin/clickhouse E 4. ./build_docker/./src/Interpreters/Context.cpp:0: DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x000000002f986fc1 in /usr/bin/clickhouse E 5. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::StorageDistributed::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x0000000032dd94d4 in /usr/bin/clickhouse E 6. ./build_docker/./src/Planner/PlannerJoinTree.cpp:756: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000003165ce97 in /usr/bin/clickhouse E 7. ./build_docker/./src/Planner/PlannerJoinTree.cpp:1450: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000003164d9aa in /usr/bin/clickhouse E 8. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x00000000315feaf6 in /usr/bin/clickhouse E 9. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000315f8e67 in /usr/bin/clickhouse E 10. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x00000000315f6ab2 in /usr/bin/clickhouse E 11. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320e467d in /usr/bin/clickhouse E 12. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320dc301 in /usr/bin/clickhouse E 13. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ecd2fd in /usr/bin/clickhouse E 14. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f0a0d0 in /usr/bin/clickhouse E 15. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcac08f in /usr/bin/clickhouse E 16. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcace57 in /usr/bin/clickhouse E 17. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e18c67c in /usr/bin/clickhouse E 18. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e185728 in /usr/bin/clickhouse E 19. ? @ 0x00007f86321e0ac3 in ? E 20. ? @ 0x00007f8632272850 in ? E . (UNKNOWN_DATABASE) E (query: SELECT * FROM remote('ch1:9000', test_default_database, test_local_table) ORDER BY i FORMAT TSV SETTINGS max_partitions_to_read = 0;) helpers/client.py:239: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ------------------------------ Captured log setup ------------------------------ 2024-09-24 00:17:12 [ 390 ] INFO : Running tests in /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/test.py (cluster.py:2637, start) 2024-09-24 00:17:12 [ 390 ] DEBUG : Cluster start called. is_up=False (cluster.py:2644, start) 2024-09-24 00:17:12 [ 390 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.2.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.2 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 17 Running: 11 Paused: 0 Stopped: 6 Images: 42 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-119-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: bae9f20c8df0 ID: 23c7ffba-fa59-4a62-b9f4-6a037892fec4 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-09-24 00:17:12 [ 390 ] DEBUG : Docker networks for project roottestpassingmaxpartitionstoreadremotely are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-09-24 00:17:12 [ 390 ] DEBUG : Docker containers for project roottestpassingmaxpartitionstoreadremotely are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-09-24 00:17:13 [ 390 ] DEBUG : Docker volumes for project roottestpassingmaxpartitionstoreadremotely are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-09-24 00:17:13 [ 390 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-09-24 00:17:13 [ 390 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.2.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.2 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 17 Running: 10 Paused: 0 Stopped: 7 Images: 42 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-119-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: bae9f20c8df0 ID: 23c7ffba-fa59-4a62-b9f4-6a037892fec4 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-09-24 00:17:13 [ 390 ] DEBUG : Docker networks for project roottestpassingmaxpartitionstoreadremotely are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-09-24 00:17:13 [ 390 ] DEBUG : Docker containers for project roottestpassingmaxpartitionstoreadremotely are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-09-24 00:17:13 [ 390 ] DEBUG : Docker volumes for project roottestpassingmaxpartitionstoreadremotely are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-09-24 00:17:13 [ 390 ] DEBUG : Command:docker container list --all --filter name='^/roottestpassingmaxpartitionstoreadremotely_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-09-24 00:17:13 [ 390 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-09-24 00:17:13 [ 390 ] DEBUG : No running containers for project: roottestpassingmaxpartitionstoreadremotely (cluster.py:782, cleanup) 2024-09-24 00:17:13 [ 390 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-09-24 00:17:13 [ 390 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-09-24 00:17:13 [ 390 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-09-24 00:17:13 [ 390 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-09-24 00:17:13 [ 390 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-09-24 00:17:13 [ 390 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-09-24 00:17:13 [ 390 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-09-24 00:17:13 [ 390 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check) 2024-09-24 00:17:13 [ 390 ] DEBUG : Setup directory for instance: ch1 (cluster.py:2657, start) 2024-09-24 00:17:13 [ 390 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/configs/config.d (cluster.py:4383, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/database (cluster.py:4400, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/logs (cluster.py:4411, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Setup directory for instance: ch2 (cluster.py:2657, start) 2024-09-24 00:17:13 [ 390 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/configs/config.d (cluster.py:4383, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/database (cluster.py:4400, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/logs (cluster.py:4411, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir) 2024-09-24 00:17:13 [ 390 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env (cluster.py:78, _create_env_file) 2024-09-24 00:17:13 [ 390 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-09-24 00:17:13 [ 390 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-09-24 00:17:13 [ 390 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-09-24 00:17:13 [ 390 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-09-24 00:17:13 [ 390 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:546, _make_request) 2024-09-24 00:17:13 [ 390 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env', '--project-name', 'roottestpassingmaxpartitionstoreadremotely', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml', 'pull'] (cluster.py:105, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch2 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch1 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch1 ... digest: sha256:3c12bcb41068a0596b... (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch1 ... status: image is up to date for a... (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch2 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch2 ... digest: sha256:3c12bcb41068a0596b... (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch2 ... status: image is up to date for a... (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : Stderr:Pulling ch2 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:22 [ 390 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env --project-name roottestpassingmaxpartitionstoreadremotely --file /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml --file /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml up -d --no-recreate') (cluster.py:2980, start) 2024-09-24 00:17:22 [ 390 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env', '--project-name', 'roottestpassingmaxpartitionstoreadremotely', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:105, run_and_check) 2024-09-24 00:17:23 [ 390 ] DEBUG : Stderr:Creating network "roottestpassingmaxpartitionstoreadremotely_default" with the default driver (cluster.py:115, run_and_check) 2024-09-24 00:17:23 [ 390 ] DEBUG : Stderr:Creating roottestpassingmaxpartitionstoreadremotely_ch2_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:23 [ 390 ] DEBUG : Stderr:Creating roottestpassingmaxpartitionstoreadremotely_ch1_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:23 [ 390 ] DEBUG : Stderr:Creating roottestpassingmaxpartitionstoreadremotely_ch1_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:23 [ 390 ] DEBUG : Stderr:Creating roottestpassingmaxpartitionstoreadremotely_ch2_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:23 [ 390 ] DEBUG : ClickHouse instance created (cluster.py:2988, start) 2024-09-24 00:17:23 [ 390 ] DEBUG : get_instance_ip instance_name=ch1 (cluster.py:1971, get_instance_ip) 2024-09-24 00:17:23 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestpassingmaxpartitionstoreadremotely_ch1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:23 [ 390 ] DEBUG : Waiting for ClickHouse start in ch1, ip: 172.16.23.2... (cluster.py:2995, start) 2024-09-24 00:17:23 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestpassingmaxpartitionstoreadremotely_ch1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:23 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:23 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:23 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:24 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/489ed72670ef7245c10bafe8639f7489971c8fd82807be3426920c603d8f49d0/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : ClickHouse ch1 started (cluster.py:2999, start) 2024-09-24 00:17:25 [ 390 ] DEBUG : get_instance_ip instance_name=ch2 (cluster.py:1971, get_instance_ip) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestpassingmaxpartitionstoreadremotely_ch2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : Waiting for ClickHouse start in ch2, ip: 172.16.23.3... (cluster.py:2995, start) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestpassingmaxpartitionstoreadremotely_ch2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/13131a2a71fe6fb7299c7085530a76d8343036e67ab8bee28c17e6dc0908c713/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : http://localhost:None "GET /v1.42/containers/13131a2a71fe6fb7299c7085530a76d8343036e67ab8bee28c17e6dc0908c713/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:17:25 [ 390 ] DEBUG : ClickHouse ch2 started (cluster.py:2999, start) 2024-09-24 00:17:25 [ 390 ] DEBUG : Executing query CREATE DATABASE test_default_database; on ch1 (cluster.py:3423, query) ------------------------------ Captured log call ------------------------------- 2024-09-24 00:17:26 [ 390 ] DEBUG : Executing query CREATE TABLE test_local_table ENGINE MergeTree PARTITION BY i ORDER BY tuple() SETTINGS max_partitions_to_read = 1 AS SELECT arrayJoin([1, 2]) i; on ch1 (cluster.py:3423, query) 2024-09-24 00:17:26 [ 390 ] DEBUG : Executing query SELECT * FROM remote('ch1:9000', test_default_database, test_local_table) ORDER BY i FORMAT TSV SETTINGS max_partitions_to_read = 0; on ch2 (cluster.py:3423, query) --------------------------- Captured stderr teardown --------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ---------------------------- Captured log teardown ----------------------------- 2024-09-24 00:17:27 [ 390 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env', '--project-name', 'roottestpassingmaxpartitionstoreadremotely', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check) 2024-09-24 00:17:30 [ 390 ] DEBUG : Stderr:Stopping roottestpassingmaxpartitionstoreadremotely_ch2_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:30 [ 390 ] DEBUG : Stderr:Stopping roottestpassingmaxpartitionstoreadremotely_ch1_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:30 [ 390 ] DEBUG : Stderr:Stopping roottestpassingmaxpartitionstoreadremotely_ch1_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:30 [ 390 ] DEBUG : Stderr:Stopping roottestpassingmaxpartitionstoreadremotely_ch2_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:30 [ 390 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:30 [ 390 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-09-24 00:17:30 [ 390 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env', '--project-name', 'roottestpassingmaxpartitionstoreadremotely', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml', 'down', '--volumes'] (cluster.py:105, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Stderr:Removing roottestpassingmaxpartitionstoreadremotely_ch2_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Stderr:Removing roottestpassingmaxpartitionstoreadremotely_ch1_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Stderr:Removing roottestpassingmaxpartitionstoreadremotely_ch2_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Stderr:Removing roottestpassingmaxpartitionstoreadremotely_ch1_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Stderr:Removing network roottestpassingmaxpartitionstoreadremotely_default (cluster.py:115, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-09-24 00:17:31 [ 390 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.2.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.2 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 18 Running: 18 Paused: 0 Stopped: 0 Images: 42 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-119-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: bae9f20c8df0 ID: 23c7ffba-fa59-4a62-b9f4-6a037892fec4 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-09-24 00:17:31 [ 390 ] DEBUG : Docker networks for project roottestpassingmaxpartitionstoreadremotely are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-09-24 00:17:31 [ 390 ] DEBUG : Docker containers for project roottestpassingmaxpartitionstoreadremotely are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-09-24 00:17:31 [ 390 ] DEBUG : Docker volumes for project roottestpassingmaxpartitionstoreadremotely are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-09-24 00:17:31 [ 390 ] DEBUG : Command:docker container list --all --filter name='^/roottestpassingmaxpartitionstoreadremotely_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-09-24 00:17:31 [ 390 ] DEBUG : No running containers for project: roottestpassingmaxpartitionstoreadremotely (cluster.py:782, cleanup) 2024-09-24 00:17:31 [ 390 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-09-24 00:17:31 [ 390 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-09-24 00:17:31 [ 390 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-09-24 00:17:31 [ 390 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-09-24 00:17:31 [ 390 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-09-24 00:17:31 [ 390 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check) __________________ test_mutations_with_merge_background_task ___________________ [gw3] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_mutations_with_merge_background_task(started_cluster): instance_test_mutations.query( """SYSTEM STOP MERGES test_mutations_with_ast_elements""" ) ## The number of asts per query is 15 for execution_times_for_mutation in range(100): instance_test_mutations.query( """ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1)""" ) all_done = False for wait_times_for_mutation in range(100): # wait for replication 80 seconds max time.sleep(0.8) def get_done_mutations(instance): instance_test_mutations.query( """DETACH TABLE test_mutations_with_ast_elements""" ) instance_test_mutations.query( """ATTACH TABLE test_mutations_with_ast_elements""" ) return int( instance.query( "SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0" ).rstrip() ) if get_done_mutations(instance_test_mutations) == 100: all_done = True break print( instance_test_mutations.query( "SELECT mutation_id, command, parts_to_do, is_done FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 FORMAT TSVWithNames" ) ) > assert all_done E assert False test_mutations_with_merge_tree/test.py:67: AssertionError ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ------------------------------ Captured log setup ------------------------------ 2024-09-24 00:16:08 [ 372 ] INFO : Running tests in /ClickHouse/tests/integration/test_mutations_with_merge_tree/test.py (cluster.py:2637, start) 2024-09-24 00:16:08 [ 372 ] DEBUG : Cluster start called. is_up=False (cluster.py:2644, start) 2024-09-24 00:16:08 [ 372 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.2.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.2 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 27 Running: 26 Paused: 0 Stopped: 1 Images: 42 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc io.containerd.runc.v2 Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-119-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: bae9f20c8df0 ID: 23c7ffba-fa59-4a62-b9f4-6a037892fec4 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-09-24 00:16:08 [ 372 ] DEBUG : Docker networks for project roottestmutationswithmergetree are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-09-24 00:16:08 [ 372 ] DEBUG : Docker containers for project roottestmutationswithmergetree are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-09-24 00:16:08 [ 372 ] DEBUG : Docker volumes for project roottestmutationswithmergetree are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-09-24 00:16:08 [ 372 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-09-24 00:16:08 [ 372 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.2.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.2 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 27 Running: 26 Paused: 0 Stopped: 1 Images: 42 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-119-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: bae9f20c8df0 ID: 23c7ffba-fa59-4a62-b9f4-6a037892fec4 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-09-24 00:16:08 [ 372 ] DEBUG : Docker networks for project roottestmutationswithmergetree are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-09-24 00:16:08 [ 372 ] DEBUG : Docker containers for project roottestmutationswithmergetree are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-09-24 00:16:08 [ 372 ] DEBUG : Docker volumes for project roottestmutationswithmergetree are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-09-24 00:16:08 [ 372 ] DEBUG : Command:docker container list --all --filter name='^/roottestmutationswithmergetree_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-09-24 00:16:08 [ 372 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-09-24 00:16:08 [ 372 ] DEBUG : No running containers for project: roottestmutationswithmergetree (cluster.py:782, cleanup) 2024-09-24 00:16:08 [ 372 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-09-24 00:16:08 [ 372 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-09-24 00:16:08 [ 372 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-09-24 00:16:08 [ 372 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-09-24 00:16:08 [ 372 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-09-24 00:16:08 [ 372 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-09-24 00:16:08 [ 372 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-09-24 00:16:08 [ 372 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check) 2024-09-24 00:16:08 [ 372 ] DEBUG : Setup directory for instance: test_mutations_with_merge_tree (cluster.py:2657, start) 2024-09-24 00:16:08 [ 372 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir) 2024-09-24 00:16:08 [ 372 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir) 2024-09-24 00:16:08 [ 372 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir) 2024-09-24 00:16:08 [ 372 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir) 2024-09-24 00:16:08 [ 372 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_mutations_with_merge_tree/configs/config.xml'] to /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/configs/config.d (cluster.py:4383, create_dir) 2024-09-24 00:16:08 [ 372 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/database (cluster.py:4400, create_dir) 2024-09-24 00:16:08 [ 372 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/logs (cluster.py:4411, create_dir) 2024-09-24 00:16:08 [ 372 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir) 2024-09-24 00:16:08 [ 372 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/.env (cluster.py:78, _create_env_file) 2024-09-24 00:16:08 [ 372 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-09-24 00:16:08 [ 372 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-09-24 00:16:08 [ 372 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-09-24 00:16:08 [ 372 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-09-24 00:16:08 [ 372 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:546, _make_request) 2024-09-24 00:16:08 [ 372 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/.env', '--project-name', 'roottestmutationswithmergetree', '--file', '/ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/docker-compose.yml', 'pull'] (cluster.py:105, run_and_check) 2024-09-24 00:16:18 [ 372 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... (cluster.py:115, run_and_check) 2024-09-24 00:16:18 [ 372 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check) 2024-09-24 00:16:18 [ 372 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... digest: sha256:3c12bcb41068a0596b... (cluster.py:115, run_and_check) 2024-09-24 00:16:18 [ 372 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... status: image is up to date for a... (cluster.py:115, run_and_check) 2024-09-24 00:16:18 [ 372 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... done (cluster.py:115, run_and_check) 2024-09-24 00:16:18 [ 372 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/.env --project-name roottestmutationswithmergetree --file /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/docker-compose.yml up -d --no-recreate') (cluster.py:2980, start) 2024-09-24 00:16:18 [ 372 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/.env', '--project-name', 'roottestmutationswithmergetree', '--file', '/ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:105, run_and_check) 2024-09-24 00:16:19 [ 372 ] DEBUG : Stderr:Creating network "roottestmutationswithmergetree_default" with the default driver (cluster.py:115, run_and_check) 2024-09-24 00:16:19 [ 372 ] DEBUG : Stderr:Creating roottestmutationswithmergetree_test_mutations_with_merge_tree_1 ... (cluster.py:115, run_and_check) 2024-09-24 00:16:19 [ 372 ] DEBUG : Stderr:Creating roottestmutationswithmergetree_test_mutations_with_merge_tree_1 ... done (cluster.py:115, run_and_check) 2024-09-24 00:16:19 [ 372 ] DEBUG : ClickHouse instance created (cluster.py:2988, start) 2024-09-24 00:16:19 [ 372 ] DEBUG : get_instance_ip instance_name=test_mutations_with_merge_tree (cluster.py:1971, get_instance_ip) 2024-09-24 00:16:19 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestmutationswithmergetree_test_mutations_with_merge_tree_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:19 [ 372 ] DEBUG : Waiting for ClickHouse start in test_mutations_with_merge_tree, ip: 172.16.13.2... (cluster.py:2995, start) 2024-09-24 00:16:19 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestmutationswithmergetree_test_mutations_with_merge_tree_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:19 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:19 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:19 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:19 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:20 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:20 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:20 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:20 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:20 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:20 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:20 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:20 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:20 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:21 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:22 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:22 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:22 [ 372 ] DEBUG : http://localhost:None "GET /v1.42/containers/3d619c05ee5d1c8c3c2ed13f8002f9688a31e1c12b8ae7529ecd7ad469da79b7/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-09-24 00:16:22 [ 372 ] DEBUG : ClickHouse test_mutations_with_merge_tree started (cluster.py:2999, start) 2024-09-24 00:16:22 [ 372 ] DEBUG : Executing query CREATE TABLE test_mutations_with_ast_elements(date Date, a UInt64, b String) ENGINE = MergeTree PARTITION BY toYYYYMM(date) ORDER BY (a, date) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:22 [ 372 ] DEBUG : Executing query INSERT INTO test_mutations_with_ast_elements SELECT '2019-07-29' AS date, 1, toString(number) FROM numbers(1) SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) ----------------------------- Captured stdout call ----------------------------- mutation_id command parts_to_do is_done mutation_2.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_3.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_4.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_5.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_6.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_7.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_8.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_9.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_10.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_11.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_12.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_13.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_14.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_15.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_16.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_17.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_18.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_19.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_20.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_21.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_22.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_23.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_24.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_25.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_26.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_27.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_28.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_29.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_30.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_31.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_32.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_33.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_34.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_35.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_36.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_37.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_38.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_39.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_40.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_41.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_42.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_43.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_44.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_45.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_46.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_47.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_48.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_49.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_50.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_51.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_52.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_53.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_54.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_55.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_56.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_57.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_58.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_59.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_60.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_61.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_62.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_63.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_64.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_65.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_66.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_67.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_68.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_69.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_70.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_71.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_72.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_73.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_74.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_75.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_76.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_77.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_78.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_79.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_80.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_81.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_82.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_83.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_84.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_85.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_86.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_87.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_88.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_89.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_90.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_91.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_92.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_93.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_94.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_95.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_96.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_97.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_98.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_99.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_100.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_101.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 ------------------------------ Captured log call ------------------------------- 2024-09-24 00:16:22 [ 372 ] DEBUG : Executing query SYSTEM STOP MERGES test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:22 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:23 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:23 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:23 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:23 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:24 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:24 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:24 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:24 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:24 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:25 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:25 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:25 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:25 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:26 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:26 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:26 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:26 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:26 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:27 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:27 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:27 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:27 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:27 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:28 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:28 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:28 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:28 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:28 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:29 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:29 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:29 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:29 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:30 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:30 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:30 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:30 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:30 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:31 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:31 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:32 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:32 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:32 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:32 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:32 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:32 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:33 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:33 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:33 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:33 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:33 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:34 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:34 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:34 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:34 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:35 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:35 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:35 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:35 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:36 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:36 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:36 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:36 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:36 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:37 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:37 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:37 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:37 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:38 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:38 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:38 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:39 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:39 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:39 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:39 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:40 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:40 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:40 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:40 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:40 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:41 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:41 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:41 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:41 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:41 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:41 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:42 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:42 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:42 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:42 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:42 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:43 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:43 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:43 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:43 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:43 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:44 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:44 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:44 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:45 [ 372 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:45 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:46 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:46 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:47 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:47 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:47 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:48 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:49 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:49 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:50 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:50 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:50 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:51 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:52 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:52 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:53 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:53 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:54 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:55 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:55 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:55 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:56 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:56 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:56 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:57 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:58 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:58 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:59 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:59 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:16:59 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:00 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:00 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:01 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:02 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:02 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:02 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:03 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:03 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:04 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:05 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:05 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:05 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:06 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:06 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:07 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:08 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:08 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:08 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:09 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:10 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:10 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:11 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:11 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:12 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:13 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:13 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:13 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:14 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:15 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:15 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:16 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:16 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:16 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:17 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:17 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:18 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:19 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:19 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:19 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:20 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:20 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:20 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:21 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:22 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:22 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:23 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:23 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:24 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:25 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:25 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:25 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:26 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:26 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:26 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:27 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:28 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:28 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:29 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:29 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:30 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:31 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:31 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:31 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:32 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:32 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:32 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:33 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:34 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:34 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:35 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:35 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:35 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:36 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:36 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:37 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:38 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:38 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:38 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:39 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:39 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:40 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:41 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:41 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:41 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:42 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:43 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:43 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:44 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:44 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:44 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:45 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:45 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:46 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:47 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:47 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:47 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:48 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:48 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:49 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:50 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:50 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:50 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:51 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:51 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:51 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:52 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:53 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:53 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:54 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:54 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:55 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:56 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:56 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:56 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:57 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:57 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:57 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:59 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:59 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:17:59 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:00 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:00 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:00 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:01 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:02 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:02 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:03 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:03 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:03 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:04 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:04 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:05 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:06 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:06 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:06 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:07 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:07 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:08 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:09 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:09 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:09 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:10 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:10 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:11 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:12 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:12 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:12 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:13 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:13 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:14 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:15 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:15 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:15 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:16 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:16 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:16 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:17 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:18 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:18 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:19 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:19 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:19 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:20 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:21 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:21 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:22 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:22 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:22 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:23 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:24 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:24 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:25 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:25 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:25 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:26 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:27 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:27 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:28 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:28 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:28 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:29 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:29 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:30 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:31 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:31 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:31 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:32 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:32 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:33 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:34 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:34 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:34 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:35 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:35 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:35 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:36 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:37 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:37 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:38 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:38 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:38 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:39 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:40 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:40 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:41 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:41 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:41 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:42 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:43 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:43 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:44 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:44 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:44 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:45 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:46 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:46 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:47 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:47 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:47 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:48 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:48 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:49 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:50 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:50 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:50 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:51 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:51 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:52 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:53 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:53 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:53 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:54 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:54 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:55 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:56 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:56 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:56 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:57 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:57 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:58 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:59 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:59 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:18:59 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:00 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:00 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:00 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:01 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:02 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:02 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:03 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:03 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:03 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:04 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:04 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:05 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:06 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:06 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:06 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:07 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:07 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:08 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:09 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:09 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:09 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:10 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:10 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:10 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:11 [ 372 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:12 [ 372 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:12 [ 372 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-09-24 00:19:12 [ 372 ] DEBUG : Executing query SELECT mutation_id, command, parts_to_do, is_done FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 FORMAT TSVWithNames on test_mutations_with_merge_tree (cluster.py:3423, query) ============================== slowest durations =============================== 170.07s call test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task 54.06s call test_mask_sensitive_info/test.py::test_backup_to_s3 45.75s call test_keeper_mntr_pressure/test.py::test_aggressive_mntr 43.58s call test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 43.47s call test_keeper_nodes_move/test.py::test_node_move 37.36s call test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 33.15s setup test_recovery_replica/test.py::test_choose_source_replica 33.07s call test_mask_sensitive_info/test.py::test_table_functions 28.28s call test_mask_sensitive_info/test.py::test_table_function_ways_to_call 27.19s setup test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch 26.49s call test_recovery_replica/test.py::test_update_metadata 24.94s setup test_reloading_storage_configuration/test.py::test_add_disk 23.59s setup test_part_uuid/test.py::test_part_uuid 23.20s teardown test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns 23.19s teardown test_log_family_s3/test.py::test_log_family_s3[TinyLog] 22.84s setup test_replace_partition/test.py::test_normal_work 22.33s setup test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with 21.88s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 21.33s setup test_mask_sensitive_info/test.py::test_backup_to_s3 20.19s setup test_reload_zookeeper/test.py::test_reload_zookeeper 19.59s call test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 19.32s setup test_keeper_four_word_command/test.py::test_cmd_clrs 19.32s setup test_log_family_s3/test.py::test_log_family_s3[Log] 18.50s call test_reload_zookeeper/test.py::test_reload_zookeeper 18.00s setup test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 17.91s call test_row_policy/test.py::test_change_of_users_xml_changes_row_policies 17.47s call test_mask_sensitive_info/test.py::test_create_table 17.20s setup test_keeper_mntr_pressure/test.py::test_aggressive_mntr 17.11s setup test_keeper_nodes_move/test.py::test_node_move 16.73s setup test_merge_table_over_distributed/test.py::test_filtering 16.72s setup test_keeper_session/test.py::test_session_close_shutdown 16.24s setup test_keeper_four_word_command/test_allow_list.py::test_allow_list 15.97s setup test_multi_access_storage_role_management/test.py::test_quotas 15.09s setup test_rocksdb_options/test.py::test_invalid_column_family_options 14.94s setup test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 14.56s setup test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task 14.53s setup test_logs_level/test.py::test_check_client_logs_level 14.24s setup test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings 13.88s call test_keeper_session/test.py::test_session_close_shutdown 13.52s setup test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 13.19s setup test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 12.84s setup test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 12.60s call test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 12.35s teardown test_reload_zookeeper/test.py::test_reload_zookeeper 12.33s setup test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 11.83s call test_rocksdb_options/test.py::test_table_invalid_options 11.81s call test_rocksdb_options/test.py::test_table_invalid_column_family_options 11.65s setup test_read_temporary_tables_on_failure/test.py::test_different_versions 11.22s call test_rocksdb_options/test.py::test_invalid_options 10.86s call test_rocksdb_options/test.py::test_invalid_column_family_options 10.75s call test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml 10.54s call test_row_policy/test.py::test_miscellaneous_engines 10.51s call test_row_policy/test.py::test_introspection 10.50s call test_row_policy/test.py::test_dcl_introspection 10.50s call test_row_policy/test.py::test_join 9.68s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 8.26s call test_reloading_storage_configuration/test.py::test_new_policy_works 8.21s call test_multi_access_storage_role_management/test.py::test_role_from_different_storages 7.99s call test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 7.80s call test_reloading_storage_configuration/test.py::test_add_volume_to_policy 7.76s call test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 7.66s call test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 7.58s teardown test_read_temporary_tables_on_failure/test.py::test_different_versions 7.46s call test_recovery_replica/test.py::test_choose_source_replica 7.38s call test_reloading_storage_configuration/test.py::test_add_disk_to_policy 7.28s call test_reloading_storage_configuration/test.py::test_remove_policy 7.27s call test_reloading_storage_configuration/test.py::test_update_disk 6.73s call test_reloading_storage_configuration/test.py::test_add_policy 6.66s call test_reloading_storage_configuration/test.py::test_remove_disk 6.42s teardown test_recovery_replica/test.py::test_update_metadata 6.23s call test_reloading_storage_configuration/test.py::test_add_disk 6.08s teardown test_part_uuid/test.py::test_part_uuid 6.02s teardown test_replace_partition/test.py::test_normal_work 5.87s call test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 5.77s teardown test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings 5.60s call test_recovery_replica/test.py::test_recovery 5.58s call test_multi_access_storage_role_management/test.py::test_settings_profiles 5.44s call test_read_temporary_tables_on_failure/test.py::test_different_versions 5.34s call test_multi_access_storage_role_management/test.py::test_roles 5.26s call test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 5.14s call test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with 4.99s call test_multi_access_storage_role_management/test.py::test_quotas 4.89s call test_mask_sensitive_info/test.py::test_create_alter_user 4.86s call test_multi_access_storage_role_management/test.py::test_users 4.74s call test_part_uuid/test.py::test_part_uuid 4.69s teardown test_mask_sensitive_info/test.py::test_table_functions 4.57s teardown test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 4.56s teardown test_row_policy/test.py::test_smoke 4.47s teardown test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 3.99s teardown test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 3.81s teardown test_logs_level/test.py::test_check_client_logs_level 3.56s teardown test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 3.44s teardown test_reloading_storage_configuration/test.py::test_update_disk 3.39s teardown test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 3.33s teardown test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 3.26s teardown test_keeper_nodes_move/test.py::test_node_move 3.22s call test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned 2.94s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 2.87s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 2.87s teardown test_keeper_session/test.py::test_session_timeout 2.82s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 2.82s teardown test_multi_access_storage_role_management/test.py::test_users 2.77s call test_row_policy/test.py::test_policy_on_distributed_table_via_role 2.76s teardown test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 2.72s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 2.72s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 2.63s call test_replace_partition/test.py::test_normal_work 2.62s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 2.62s call test_mask_sensitive_info/test.py::test_encryption_functions 2.59s teardown test_keeper_four_word_command/test_allow_list.py::test_allow_list 2.51s teardown test_keeper_mntr_pressure/test.py::test_aggressive_mntr 2.47s teardown test_keeper_four_word_command/test.py::test_cmd_wchs 2.28s call test_mask_sensitive_info/test.py::test_create_dictionary 2.26s call test_log_family_s3/test.py::test_log_family_s3[StripeLog] 2.21s call test_mask_sensitive_info/test.py::test_create_database 2.19s teardown test_rocksdb_options/test.py::test_valid_options 2.12s call test_mask_sensitive_info/test.py::test_on_cluster 2.12s teardown test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 2.07s teardown test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with 2.01s call test_log_family_s3/test.py::test_log_family_s3[Log] 2.01s call test_log_family_s3/test.py::test_log_family_s3[TinyLog] 1.82s call test_row_policy/test.py::test_smoke 1.51s call test_keeper_four_word_command/test_allow_list.py::test_allow_list 1.50s call test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch 1.30s call test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns 1.27s call test_rocksdb_options/test.py::test_valid_options 1.16s call test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings 1.04s call test_keeper_four_word_command/test.py::test_cmd_rqld 0.80s call test_keeper_four_word_command/test.py::test_cmd_mntr 0.75s call test_merge_table_over_distributed/test.py::test_filtering 0.70s call test_keeper_four_word_command/test.py::test_cmd_lgif 0.65s call test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 0.55s call test_keeper_session/test.py::test_session_timeout 0.53s call test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 0.48s call test_merge_table_over_distributed/test.py::test_global_in 0.42s call test_rocksdb_options/test.py::test_table_valid_options 0.39s call test_keeper_four_word_command/test.py::test_cmd_stat 0.34s call test_keeper_four_word_command/test.py::test_cmd_srvr 0.30s call test_keeper_four_word_command/test.py::test_cmd_crst 0.29s call test_keeper_four_word_command/test.py::test_cmd_cons 0.29s call test_rocksdb_options/test.py::test_table_valid_column_family_options 0.28s call test_keeper_four_word_command/test.py::test_cmd_dump 0.28s call test_keeper_four_word_command/test.py::test_cmd_wchc 0.26s call test_keeper_four_word_command/test.py::test_cmd_wchp 0.24s call test_keeper_four_word_command/test.py::test_cmd_wchs 0.23s call test_rocksdb_options/test.py::test_valid_column_family_options 0.22s call test_keeper_four_word_command/test.py::test_cmd_clrs 0.22s call test_logs_level/test.py::test_check_client_logs_level 0.15s call test_keeper_four_word_command/test.py::test_cmd_srst 0.13s call test_keeper_four_word_command/test.py::test_cmd_conf 0.13s call test_keeper_four_word_command/test.py::test_cmd_csnp 0.13s teardown test_row_policy/test.py::test_change_of_users_xml_changes_row_policies 0.03s teardown test_row_policy/test.py::test_join 0.02s teardown test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml 0.02s teardown test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned 0.02s teardown test_row_policy/test.py::test_grant_create_row_policy 0.02s call test_row_policy/test.py::test_grant_create_row_policy 0.02s call test_row_policy/test.py::test_dcl_management 0.02s teardown test_row_policy/test.py::test_row_policy_filter_with_subquery 0.02s call test_row_policy/test.py::test_row_policy_filter_with_subquery 0.02s teardown test_row_policy/test.py::test_dcl_management 0.02s teardown test_row_policy/test.py::test_introspection 0.02s teardown test_row_policy/test.py::test_miscellaneous_engines 0.02s teardown test_row_policy/test.py::test_policy_on_distributed_table_via_role 0.02s teardown test_row_policy/test.py::test_dcl_introspection 0.02s call test_keeper_four_word_command/test.py::test_cmd_isro 0.02s call test_keeper_four_word_command/test.py::test_cmd_ruok 0.02s teardown test_row_policy/test.py::test_reload_users_xml_by_timer 0.02s call test_row_policy/test.py::test_reload_users_xml_by_timer 0.00s setup test_row_policy/test.py::test_dcl_introspection 0.00s teardown test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch 0.00s teardown test_recovery_replica/test.py::test_choose_source_replica 0.00s teardown test_mask_sensitive_info/test.py::test_create_alter_user 0.00s setup test_row_policy/test.py::test_row_policy_filter_with_subquery 0.00s teardown test_mask_sensitive_info/test.py::test_backup_to_s3 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk 0.00s setup test_row_policy/test.py::test_grant_create_row_policy 0.00s setup test_row_policy/test.py::test_introspection 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 0.00s setup test_row_policy/test.py::test_policy_on_distributed_table_via_role 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_mntr 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchp 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_policy 0.00s setup test_row_policy/test.py::test_join 0.00s setup test_row_policy/test.py::test_dcl_management 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[Log] 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_lgif 0.00s setup test_mask_sensitive_info/test.py::test_table_function_ways_to_call 0.00s setup test_mask_sensitive_info/test.py::test_create_database 0.00s setup test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml 0.00s setup test_log_family_s3/test.py::test_log_family_s3[TinyLog] 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.00s setup test_multi_access_storage_role_management/test.py::test_settings_profiles 0.00s teardown test_mask_sensitive_info/test.py::test_create_table 0.00s setup test_recovery_replica/test.py::test_recovery 0.00s setup test_recovery_replica/test.py::test_update_metadata 0.00s setup test_rocksdb_options/test.py::test_valid_options 0.00s setup test_row_policy/test.py::test_miscellaneous_engines 0.00s setup test_row_policy/test.py::test_change_of_users_xml_changes_row_policies 0.00s teardown test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s setup test_row_policy/test.py::test_reload_users_xml_by_timer 0.00s setup test_keeper_four_word_command/test.py::test_cmd_srvr 0.00s setup test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_stat 0.00s setup test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srvr 0.00s setup test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_clrs 0.00s setup test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s setup test_keeper_four_word_command/test.py::test_cmd_rqld 0.00s teardown test_multi_access_storage_role_management/test.py::test_settings_profiles 0.00s setup test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns 0.00s setup test_row_policy/test.py::test_smoke 0.00s setup test_merge_table_over_distributed/test.py::test_global_in 0.00s setup test_rocksdb_options/test.py::test_valid_column_family_options 0.00s setup test_keeper_four_word_command/test.py::test_cmd_stat 0.00s setup test_multi_access_storage_role_management/test.py::test_users 0.00s teardown test_rocksdb_options/test.py::test_invalid_options 0.00s teardown test_multi_access_storage_role_management/test.py::test_role_from_different_storages 0.00s teardown test_rocksdb_options/test.py::test_valid_column_family_options 0.00s setup test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_options 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_crst 0.00s setup test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_dump 0.00s setup test_mask_sensitive_info/test.py::test_create_alter_user 0.00s teardown test_keeper_session/test.py::test_session_close_shutdown 0.00s teardown test_mask_sensitive_info/test.py::test_encryption_functions 0.00s setup test_keeper_four_word_command/test.py::test_cmd_ruok 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchc 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s teardown test_multi_access_storage_role_management/test.py::test_roles 0.00s setup test_keeper_four_word_command/test.py::test_cmd_mntr 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchs 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchp 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srst 0.00s teardown test_merge_table_over_distributed/test.py::test_global_in 0.00s setup test_keeper_four_word_command/test.py::test_cmd_srst 0.00s setup test_keeper_four_word_command/test.py::test_cmd_isro 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s setup test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s setup test_mask_sensitive_info/test.py::test_table_functions 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_cons 0.00s teardown test_recovery_replica/test.py::test_recovery 0.00s setup test_keeper_four_word_command/test.py::test_cmd_conf 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_rqld 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchc 0.00s teardown test_multi_access_storage_role_management/test.py::test_quotas 0.00s setup test_multi_access_storage_role_management/test.py::test_role_from_different_storages 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s setup test_keeper_four_word_command/test.py::test_cmd_dump 0.00s setup test_keeper_four_word_command/test.py::test_cmd_csnp 0.00s teardown test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s teardown test_rocksdb_options/test.py::test_table_valid_options 0.00s setup test_keeper_four_word_command/test.py::test_cmd_crst 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_csnp 0.00s setup test_mask_sensitive_info/test.py::test_create_table 0.00s setup test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s teardown test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_ruok 0.00s setup test_reloading_storage_configuration/test.py::test_update_disk 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.00s setup test_rocksdb_options/test.py::test_table_invalid_options 0.00s setup test_mask_sensitive_info/test.py::test_on_cluster 0.00s setup test_keeper_session/test.py::test_session_timeout 0.00s teardown test_mask_sensitive_info/test.py::test_create_dictionary 0.00s teardown test_merge_table_over_distributed/test.py::test_filtering 0.00s setup test_keeper_four_word_command/test.py::test_cmd_lgif 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.00s setup test_multi_access_storage_role_management/test.py::test_roles 0.00s setup test_keeper_four_word_command/test.py::test_cmd_cons 0.00s teardown test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s setup test_mask_sensitive_info/test.py::test_encryption_functions 0.00s setup test_reloading_storage_configuration/test.py::test_remove_policy 0.00s setup test_reloading_storage_configuration/test.py::test_add_policy 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_conf 0.00s setup test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s setup test_rocksdb_options/test.py::test_invalid_options 0.00s setup test_mask_sensitive_info/test.py::test_create_dictionary 0.00s teardown test_reloading_storage_configuration/test.py::test_add_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_isro 0.00s setup test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 0.00s teardown test_mask_sensitive_info/test.py::test_table_function_ways_to_call 0.00s teardown test_mask_sensitive_info/test.py::test_create_database 0.00s teardown test_mask_sensitive_info/test.py::test_on_cluster =========================== short test summary info ============================ FAILED test_merge_table_over_distributed/test.py::test_global_in - helpers.cl... FAILED test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed FAILED test_row_policy/test.py::test_change_of_users_xml_changes_row_policies FAILED test_row_policy/test.py::test_dcl_introspection - helpers.client.Query... FAILED test_row_policy/test.py::test_dcl_management - Exception: Command ['do... FAILED test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml FAILED test_row_policy/test.py::test_grant_create_row_policy - Exception: Com... FAILED test_row_policy/test.py::test_introspection - helpers.client.QueryRunt... FAILED test_row_policy/test.py::test_join - helpers.client.QueryRuntimeExcept... FAILED test_row_policy/test.py::test_miscellaneous_engines - helpers.client.Q... FAILED test_mask_sensitive_info/test.py::test_encryption_functions - UnicodeD... FAILED test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned FAILED test_row_policy/test.py::test_policy_on_distributed_table_via_role - h... FAILED test_row_policy/test.py::test_reload_users_xml_by_timer - Exception: C... FAILED test_row_policy/test.py::test_row_policy_filter_with_subquery - Except... FAILED test_row_policy/test.py::test_smoke - helpers.client.QueryRuntimeExcep... FAILED test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster FAILED test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task ERROR test_row_policy/test.py::test_change_of_users_xml_changes_row_policies ERROR test_row_policy/test.py::test_dcl_introspection - Exception: Command ['... ERROR test_row_policy/test.py::test_dcl_management - Exception: Command ['doc... ERROR test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml - ... ERROR test_row_policy/test.py::test_grant_create_row_policy - Exception: Comm... ERROR test_row_policy/test.py::test_introspection - Exception: Command ['dock... ERROR test_row_policy/test.py::test_join - Exception: Command ['docker', 'exe... ERROR test_row_policy/test.py::test_miscellaneous_engines - Exception: Comman... ERROR test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned ERROR test_row_policy/test.py::test_policy_on_distributed_table_via_role - Ex... ERROR test_row_policy/test.py::test_reload_users_xml_by_timer - Exception: Co... ERROR test_row_policy/test.py::test_row_policy_filter_with_subquery - Excepti... ERROR test_row_policy/test.py::test_smoke - Exception: Command ['docker', 'ex... PASSED test_merge_table_over_distributed/test.py::test_filtering PASSED test_keeper_four_word_command/test.py::test_cmd_clrs PASSED test_keeper_four_word_command/test.py::test_cmd_conf PASSED test_keeper_four_word_command/test.py::test_cmd_cons PASSED test_keeper_four_word_command/test.py::test_cmd_crst PASSED test_keeper_four_word_command/test.py::test_cmd_csnp PASSED test_keeper_four_word_command/test.py::test_cmd_dump PASSED test_keeper_four_word_command/test.py::test_cmd_isro PASSED test_multi_access_storage_role_management/test.py::test_quotas PASSED test_log_family_s3/test.py::test_log_family_s3[Log] PASSED test_keeper_four_word_command/test.py::test_cmd_lgif PASSED test_keeper_four_word_command/test.py::test_cmd_mntr PASSED test_keeper_four_word_command/test.py::test_cmd_rqld PASSED test_keeper_four_word_command/test.py::test_cmd_ruok PASSED test_keeper_four_word_command/test.py::test_cmd_srst PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] PASSED test_keeper_four_word_command/test.py::test_cmd_srvr PASSED test_keeper_four_word_command/test.py::test_cmd_stat PASSED test_keeper_four_word_command/test.py::test_cmd_wchc PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] PASSED test_keeper_four_word_command/test.py::test_cmd_wchp PASSED test_keeper_four_word_command/test.py::test_cmd_wchs PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] PASSED test_rocksdb_options/test.py::test_invalid_column_family_options PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] PASSED test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with PASSED test_multi_access_storage_role_management/test.py::test_role_from_different_storages PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] PASSED test_reloading_storage_configuration/test.py::test_add_disk PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] PASSED test_multi_access_storage_role_management/test.py::test_roles PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] PASSED test_rocksdb_options/test.py::test_invalid_options PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] PASSED test_multi_access_storage_role_management/test.py::test_settings_profiles PASSED test_recovery_replica/test.py::test_choose_source_replica PASSED test_multi_access_storage_role_management/test.py::test_users PASSED test_recovery_replica/test.py::test_recovery PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options PASSED test_reloading_storage_configuration/test.py::test_add_policy PASSED test_keeper_session/test.py::test_session_close_shutdown PASSED test_keeper_session/test.py::test_session_timeout PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy PASSED test_rocksdb_options/test.py::test_table_invalid_options PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options PASSED test_rocksdb_options/test.py::test_table_valid_options PASSED test_rocksdb_options/test.py::test_valid_column_family_options PASSED test_rocksdb_options/test.py::test_valid_options PASSED test_reloading_storage_configuration/test.py::test_new_policy_works PASSED test_recovery_replica/test.py::test_update_metadata PASSED test_mask_sensitive_info/test.py::test_backup_to_s3 PASSED test_reloading_storage_configuration/test.py::test_remove_disk PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery PASSED test_mask_sensitive_info/test.py::test_create_alter_user PASSED test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings PASSED test_mask_sensitive_info/test.py::test_create_database PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy PASSED test_mask_sensitive_info/test.py::test_create_dictionary PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns PASSED test_reloading_storage_configuration/test.py::test_remove_policy PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy PASSED test_mask_sensitive_info/test.py::test_create_table PASSED test_reloading_storage_configuration/test.py::test_update_disk PASSED test_mask_sensitive_info/test.py::test_on_cluster PASSED test_part_uuid/test.py::test_part_uuid PASSED test_keeper_nodes_move/test.py::test_node_move PASSED test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit PASSED test_replication_without_zookeeper/test.py::test_startup_without_zookeeper PASSED test_logs_level/test.py::test_check_client_logs_level PASSED test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery PASSED test_mask_sensitive_info/test.py::test_table_function_ways_to_call PASSED test_replace_partition/test.py::test_normal_work PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr PASSED test_mask_sensitive_info/test.py::test_table_functions PASSED test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table PASSED test_reload_zookeeper/test.py::test_reload_zookeeper ============= 18 failed, 82 passed, 13 errors in 309.51s (0:05:09) ============= Traceback (most recent call last): File "/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration/./runner", line 455, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_zqgugc --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_BASE_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_KERBERIZED_HADOOP_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_KERBEROS_KDC_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_MYSQL_JAVA_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_MYSQL_JS_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_MYSQL_PHP_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_NGINX_DAV_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_NEW_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_logs_level/test.py::test_check_client_logs_level test_mask_sensitive_info/test.py::test_backup_to_s3 test_mask_sensitive_info/test.py::test_create_alter_user test_mask_sensitive_info/test.py::test_create_database test_mask_sensitive_info/test.py::test_create_dictionary test_mask_sensitive_info/test.py::test_create_table test_mask_sensitive_info/test.py::test_encryption_functions test_mask_sensitive_info/test.py::test_on_cluster test_mask_sensitive_info/test.py::test_table_function_ways_to_call test_mask_sensitive_info/test.py::test_table_functions test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery test_merge_table_over_distributed/test.py::test_filtering test_merge_table_over_distributed/test.py::test_global_in test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_zookeeper/test.py::test_reload_zookeeper test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replace_partition/test.py::test_normal_work test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke -vvv" altinityinfra/integration-tests-runner:0-9992f5eba3591142ff413d31cb83a7bf4f0e2553 ' returned non-zero exit status 1.